Skip to main content

Test Quarantining

Once you've detected flaky tests, the next step is to fix them. However, you may have other priorities - here lies a dilemma. Flaky tests impact developer time, but there are other deliverables as well.

BuildPulse enables you to temporarily disable tests that flake beyond a configured threshold, to mitigate the impact of flaky tests. You can actually save the time consumed on our dashboard - this is done by:

  1. Configuring your Flakiness Threshold
  2. Querying the Quarantine API for flaky tests
  3. Disabling Tests
  4. Tracking & Re-enabling tests

Configuring your Flakiness Threshold​

To configure your flaky test threshold:

  1. Navigate to the BuildPulse Flaky Test Dashboard.
  2. Click on top-right dropdown, and navigate to "Settings" > "Repository Settings" > Flaky Tests.
  3. Set the disruption percentage which will determine quarantine status for a flaky test.

Querying the Quarantine API​

  1. Email support@buildpulse.io for your API key.
  2. Query the API for flaky tests:

URL: https://api.buildpulse.io/v1/flaky/tests

Authorization: Authorization: Bearer ${BUILDPULSE_API_KEY}

Query Parameters:

  • repository (required, or repository_id): The name of your repository/project
  • repository_id (required, or repository): The ID of your repository/project. You can find if you navigate to the repository > Flaky Tests tab > View Setup Guide (bottom left) > Other Providers > Step 2 (see Repository ID).
  • quarantine: Set to false to return all flaky tests (default: true)
  • include: Expand flaky test attributes. Supported values: tags, disruptiveness_ratio, nondeterministic_negative_result_count. Example: include=tags,disruptiveness_ratio
  • after: Pagination key. Value will be null on the last page.
  1. Sample Response:
{
    "count": 1,
    "tests": [
        {
            "id": 434140152,
            "name": "System Posts Lifecycle Basic posts flow can edit a post",
            "suite": "rspec",
            "class": "spec.system.posts_lifecycle_spec",
            "file": "./spec/system/posts_lifecycle_spec.rb"
            "disruptor_type": "default" # 'default' for git-based, 'statistical' for statistical flaky tests, 'disruptor_type_manual' for manually quarantined tests
        }
    ],
    "metadata": {
        "after": null,
        "limit": 25
    }
}

Disabling Tests​

Please consult the documentation of your test framework for disabling individual tests.

Tracking & Re-enabling tests​

BuildPulse will preserve quarantined tests beyond the data retention period, and are tagged as 'quarantined' in the BuildPulse Flaky Test Dashboard.

When the tests is resolved, you can re-enable tests by removing the quarantine tag from the test name in the dashboard. This will remove the test from the quarantined list, and will no longer be ignored by your test framework.